산업첓 ģž¬ģ§ģž ėŒ€ģƒ ģøź³µģ§€ėŠ„ 핵심 기쓈 교윔

Predicting the Weight of Injection Molding Product


Prof. Seungchul Lee
http://iai.postech.ac.kr/
Industrial AI Lab at POSTECH

Table of Contents

InĀ [1]:
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
InĀ [2]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf
import seaborn as sns
import random
import os

1. Understand Machine Learning and Deep Learning Models¶

1.1. Linear Regression¶

Regression analysis is the process of finding the function $f(x)$ that outputs the most similar value $\hat{y}$ to the dependent variable $y$ corresponding to the independent variable $x$.

$$ \hat{y} = f(x) ā‰ˆ y$$

If $f(x)$ is a linear function, then this function is called a linear regression model.

$$ \hat{y} = \omega_0 + \omega_1x_1 + \omega_2x_2 + ... + \omega_Dx_D = \omega_0 + \omega^Tx$$

In the above equation, the independent variable $x = (x_1, x_2, ... , x_D)$ is a $D$-dimension vector. The weight vector $\omega = (\omega_0, ... , \omega_D)$ is the coefficient of the function $f(x)$ and the parameter is this linear regression model.



1.2. Decision Tree¶

  • A decision tree is a flowchart-like structure in which each internal node represents a "test" on an attribute (e.g. whether a coin flip comes up heads or tails), each branch represents the outcome of the test, and each leaf node represents a class label (decision taken after computing all attributes). The paths from root to leaf represent classification rules.







1.3. Random Forest¶

  • Ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees



1.4. Deep Neural Network¶

  • Complex/Nonlinear universal function approximator
    • Linearly connected networks
    • Simple nonlinear neurons

Multi-neurons





Differentiable activation function




In a compact representation



Multi-layer perceptron



  • Hidden layers
    • Autonomous feature learning




2. Build and Predict Machine Learning and Deep Learning Models¶

2.1. Data Description¶

  • Injection molding dataset consisting of 36 mold shapes
  • This dataset consists of 5 process features (control parameters), 32 mold shape features, and the weight of the molded part (ground truth).


InĀ [3]:
train_dataset = pd.read_excel('/content/drive/MyDrive/linc/data_files/train_dataset.xlsx', 
                              index_col = 0, 
                              engine = 'openpyxl')
train_dataset
Out[3]:
Fill time (sec) Melt temperature (ā„ƒ) Mold temperature (ā„ƒ) Packing pressure (MPa) Packing time (sec) OverallProjectionAreaZX MinGateHydraulicDiameter CavityVolume StdFlowLength RunnerSurfaceArea ... NumberOfCavities AvgPartThickness AvgGateHydraulicDiameter Overall_Compactness Cavity_Compactness Runner_Compactness CavitySurfaceToVolume RunnerSurfaceToVolume OverallSurfaceToVolume Weight (g)
MoldNumber
1 0.7 232 31 43 2.7 19.686 0.55 14.804 0.601 16.797 ... 2 1.253 0.813 1.398630 1.323422 2.400429 15.112335 8.331845 14.299703 13.7819
1 0.5 220 40 51 1.3 19.686 0.55 14.804 0.601 16.797 ... 2 1.253 0.813 1.398630 1.323422 2.400429 15.112335 8.331845 14.299703 13.1317
1 1.4 221 51 46 1.3 19.686 0.55 14.804 0.601 16.797 ... 2 1.253 0.813 1.398630 1.323422 2.400429 15.112335 8.331845 14.299703 13.3700
1 0.8 234 34 61 2.0 19.686 0.55 14.804 0.601 16.797 ... 2 1.253 0.813 1.398630 1.323422 2.400429 15.112335 8.331845 14.299703 13.5909
1 1.1 239 44 37 2.6 19.686 0.55 14.804 0.601 16.797 ... 2 1.253 0.813 1.398630 1.323422 2.400429 15.112335 8.331845 14.299703 13.7390
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
35 1.5 225 30 70 2.0 77.579 0.98 14.441 0.259 39.768 ... 2 1.154 1.011 1.352616 1.188687 2.354154 16.825289 8.495621 14.786163 16.1268
35 1.5 225 30 70 3.0 77.579 0.98 14.441 0.259 39.768 ... 2 1.154 1.011 1.352616 1.188687 2.354154 16.825289 8.495621 14.786163 16.4245
35 1.5 240 45 30 1.0 77.579 0.98 14.441 0.259 39.768 ... 2 1.154 1.011 1.352616 1.188687 2.354154 16.825289 8.495621 14.786163 15.4136
35 1.5 240 45 30 2.0 77.579 0.98 14.441 0.259 39.768 ... 2 1.154 1.011 1.352616 1.188687 2.354154 16.825289 8.495621 14.786163 15.8083
35 1.5 240 45 30 3.0 77.579 0.98 14.441 0.259 39.768 ... 2 1.154 1.011 1.352616 1.188687 2.354154 16.825289 8.495621 14.786163 16.1344

3500 rows Ɨ 38 columns

InĀ [4]:
train_dataset.describe()
Out[4]:
Fill time (sec) Melt temperature (ā„ƒ) Mold temperature (ā„ƒ) Packing pressure (MPa) Packing time (sec) OverallProjectionAreaZX MinGateHydraulicDiameter CavityVolume StdFlowLength RunnerSurfaceArea ... NumberOfCavities AvgPartThickness AvgGateHydraulicDiameter Overall_Compactness Cavity_Compactness Runner_Compactness CavitySurfaceToVolume RunnerSurfaceToVolume OverallSurfaceToVolume Weight (g)
count 3500.000000 3500.000000 3500.000000 3500.000000 3500.000000 3500.000000 3500.000000 3500.000000 3500.000000 3500.000000 ... 3500.000000 3500.000000 3500.00000 3500.000000 3500.000000 3500.000000 3500.000000 3500.000000 3500.000000 3500.000000
mean 1.002686 224.865714 45.266000 49.784000 2.018086 31.048743 1.134171 60.416314 2.088086 48.864543 ... 2.085714 2.159286 1.19860 2.254255 2.197544 1.593867 10.974949 22.288319 10.008039 51.881052
std 0.342842 10.043360 9.865042 13.314482 0.673256 27.991635 0.529301 46.290953 6.463764 57.341880 ... 1.360266 0.888018 0.66395 0.801692 0.892460 0.944723 5.255846 16.967633 3.424420 37.638686
min 0.500000 210.000000 30.000000 30.000000 0.500000 3.265000 0.319000 5.932000 0.000000 0.129000 ... 1.000000 0.725000 0.57600 1.086974 0.758980 0.447284 4.462667 6.053965 4.559499 9.887500
25% 0.700000 216.000000 37.000000 38.000000 1.400000 9.128000 0.924000 28.069000 0.000000 0.626000 ... 1.000000 1.418000 0.92400 1.556347 1.464589 0.465116 6.766880 8.495621 6.847776 22.670925
50% 1.000000 225.000000 45.000000 50.000000 2.000000 20.782000 0.980000 49.139000 0.153000 22.034000 ... 1.000000 1.956000 1.00200 1.960123 1.960405 2.026478 10.201973 9.869339 10.203440 45.398450
75% 1.300000 234.000000 54.000000 61.000000 2.600000 45.083000 1.482000 78.400000 0.601000 79.879000 ... 4.000000 2.899000 1.48200 2.920656 2.955572 2.354154 13.655710 43.000000 12.850604 59.060725
max 1.500000 240.000000 60.000000 70.000000 3.000000 125.475000 3.578000 207.283000 37.500000 191.761000 ... 5.000000 3.960000 4.69800 4.386447 4.481625 3.303620 26.351158 44.714286 18.399706 182.257600

8 rows Ɨ 38 columns

2.2. Construct Train and Test Dataset¶

  • 1 ~ 35 geometry mold: train dataset
  • Test mold: test dataset
InĀ [5]:
train_x = train_dataset.drop('Weight (g)', axis = 1)
train_y = train_dataset.loc[:, 'Weight (g)']

test_dataset = pd.read_excel('/content/drive/MyDrive/linc/data_files/test_dataset.xlsx', 
                             index_col = 0, 
                             engine = 'openpyxl')
test_x = test_dataset.drop('Weight (g)', axis = 1)
test_y = test_dataset.loc[:, 'Weight (g)']

print('train_x: {}, train_y: {}'.format(train_x.shape, train_y.shape))
print('test_x: {}, test_y: {}'.format(test_x.shape, test_y.shape))
train_x: (3500, 37), train_y: (3500,)
test_x: (100, 37), test_y: (100,)

2.3. Decision Tree¶

  • max_depth: the maximum depth of the decision tree,
    when default = None, divide until the class value is perfectly determined.
InĀ [6]:
from sklearn.tree import DecisionTreeRegressor

reg = DecisionTreeRegressor(max_depth = 10000, random_state = 42).fit(train_x, train_y)
pred_dt = reg.predict(test_x)
   
def plot_(pred, test_y, name, size = True):
    if size == True:
        plt.figure(figsize = (8, 6))
    plt.plot(pred, 'ro--', label = 'Prediction')
    plt.plot(np.array(test_y), 'bo--', label = 'Ground Truth')
    plt.legend(fontsize = 15)
    plt.ylabel('Weight (g)', fontsize = 20)
    plt.title('{}'.format(name), fontsize = 30)
    plt.xticks(fontsize = 15)
    plt.yticks(fontsize = 15)
    
plot_(pred_dt, test_y, 'Decision Tree')

2.4. Random Forest¶

  • n_estimators: How many trees to ensemble ?
InĀ [7]:
from sklearn.ensemble import RandomForestRegressor

reg = RandomForestRegressor(n_estimators = 100, 
                            max_depth = 10000, 
                            random_state = 42).fit(train_x, train_y)
pred_rf = reg.predict(test_x)

plot_(pred_rf, test_y, 'Random Forest')

2.5. Linear Regression Modeling¶

InĀ [8]:
from sklearn.linear_model import LinearRegression

reg = LinearRegression().fit(train_x, train_y)

print('Regression Coefficient: \n{}\n'.format(reg.coef_))
print('Regression Bias: \n{}'.format(reg.intercept_))
Regression Coefficient: 
[ 7.71710835e-01 -3.19812614e-02 -2.49243899e-02  3.31152425e-03
  7.44199836e-01 -1.18719384e-01 -1.39731152e+02 -3.50739358e+03
  1.58253392e+01 -2.19842350e+03 -2.19842406e+03 -2.97184154e+02
  9.53226836e+00  4.44332598e+01 -3.96366900e-01 -2.32062972e+00
 -3.26819880e+01 -8.69045736e-01 -3.74451683e-02 -7.21563953e+00
  2.19844363e+03  3.50826044e+03 -3.50712687e+03 -5.89890297e+01
  1.91988137e+02 -1.16484245e+01 -2.10800016e-01  2.17656934e+00
  2.81864581e+00 -4.65071586e+00 -4.45489515e+01 -3.95507162e+01
  4.18233457e+01  7.86503291e+00  1.97037478e+00  3.47124091e-01
 -2.35971505e+00]

Regression Bias: 
-23.15204255446134
InĀ [9]:
pred_lr = reg.predict(test_x)
pred_lr
Out[9]:
array([28.73689182, 27.45943159, 27.85232047, 28.97871944, 29.27536825,
       29.82853293, 29.32334135, 29.7689617 , 29.11690179, 28.85082292,
       28.10863057, 29.22590309, 29.06674493, 28.31248811, 29.87379821,
       28.09077154, 29.37008561, 28.55386492, 28.92239827, 28.73599575,
       27.9104892 , 28.06897856, 28.59764758, 28.32907063, 28.5403697 ,
       28.57227216, 28.34471971, 27.99995204, 29.28922192, 28.56464774,
       28.88345475, 28.46864172, 28.72691885, 28.72231828, 28.90968127,
       28.68449853, 29.57019488, 28.54721663, 28.77939432, 30.24654283,
       29.32049995, 29.46878455, 29.32209367, 27.73418205, 28.99499946,
       29.29748179, 29.05209479, 28.51578413, 29.87678766, 28.80247587,
       27.54602986, 28.44057129, 28.72179073, 28.30980943, 28.18470521,
       29.75556879, 27.73003314, 30.07298571, 28.29758293, 28.66809105,
       28.97864665, 28.29041079, 28.36312169, 29.33915134, 29.96363876,
       28.57069275, 28.00265501, 28.88283972, 28.46110322, 28.29828081,
       29.08656903, 29.23661205, 28.8327277 , 28.45849813, 29.20269797,
       29.9468978 , 27.67114385, 28.41534368, 29.15954352, 26.88378956,
       27.6279894 , 28.37218923, 28.60294867, 29.3471485 , 30.09134834,
       27.61690293, 28.36110277, 29.1053026 , 27.95114619, 28.69534603,
       29.43954586, 28.54870775, 29.29290759, 30.03710743, 28.88295101,
       29.62715085, 30.37135068, 27.89690527, 28.64110511, 29.38530495])
InĀ [10]:
plot_(pred_lr, test_y, 'Linear Regression')

3. Understand Potential Problems in Industrial Data¶

3.1. Curse of Dimensionality¶

Unlike the toy example (e.g., MNIST dataset), when applying artificial intelligence to industrial applications, a variety of issues may arise. The curse of dimensionality is an example of one of them. Because several physical phenomena overlap in the case of industrial data, it is difficult to know important features for predicted values from the standpoint of domain knowledge.



To ensure statistical stability, the number of features should be chosen to correspond to the number of data points.

Therefore, we try correlation analysis to select important features in data-based prediction.

3.2. Correlation Analysis¶

  • Which features are statistically significant?
    • Correlation analysis: a statistical analysis method that examines the relationship (closeness) of variables.
  • Analyze features that are highly related to "the weight of the injected product".


InĀ [11]:
# ģ‚¼ź°ķ˜• 마스크넼 ė§Œė“ ė‹¤(ģœ„ ģŖ½ ģ‚¼ź°ķ˜•ģ— True, ģ•„ėž˜ ģ‚¼ź°ķ˜•ģ— False)
mask = np.zeros_like(train_dataset.corr(), dtype = np.bool)
mask[np.triu_indices_from(mask)] = True

plt.figure(figsize = (24, 20))
sns.heatmap(train_dataset.corr(), 
            cmap = 'RdYlBu_r', 
            annot = True,               # ģ‹¤ģ œ ź°’ģ„ ķ‘œģ‹œķ•œė‹¤
            mask = mask,                # ķ‘œģ‹œķ•˜ģ§€ ģ•Šģ„ 마스크 ė¶€ė¶„ģ„ ģ§€ģ •ķ•œė‹¤
            linewidths = 1,             # 경계멓 ģ‹¤ģ„ ģœ¼ė”œ źµ¬ė¶„ķ•˜źø°
            cbar_kws = {"shrink": .5},  # ģ»¬ėŸ¬ė°” 크기 절반으딜 ģ¤„ģ“źø°
            vmin = -1, vmax = 1         # ģ»¬ėŸ¬ė°” ė²”ģœ„ -1 ~ 1
           )  
plt.show()
<ipython-input-11-ab7667a5af00>:2: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  mask = np.zeros_like(train_dataset.corr(), dtype=np.bool)
InĀ [12]:
weight_corr = train_dataset.corr()['Weight (g)']
weight_corr.sort_values(ascending = False)
Out[12]:
Weight (g)                  1.000000
OverallVolume               0.998736
CavityVolume                0.985415
CavitySurfaceArea           0.858996
OverallSurfaceArea          0.856717
OverallProjectionAreaXY     0.729818
OverallProjectionAreaYZ     0.556189
MaxPartThickness            0.514668
OverallProjectionAreaZX     0.485988
Cavity_Compactness          0.484543
Overall_Compactness         0.479901
AvgFlowLength               0.434925
StdPartThickness            0.422464
MaxFlowLength               0.420291
MinFlowLength               0.416272
AvgPartThickness            0.404082
RunnerVolume                0.355200
RunnerSurfaceArea           0.270440
MaxFlowLengthToThickness    0.216423
AvgFlowLengthToThickness    0.207084
MinFlowLengthToThickness    0.199110
MinGateHydraulicDiameter    0.132351
AvgGateHydraulicDiameter    0.094403
StdFlowLength               0.081371
Runner_Compactness          0.069361
MaxGateHydraulicDiameter    0.061674
StdFlowLengthToThickness    0.030716
RunnerSurfaceToVolume       0.021096
Packing pressure (MPa)      0.009901
Packing time (sec)         -0.004789
Fill time (sec)            -0.011950
Mold temperature (ā„ƒ)       -0.016511
Melt temperature (ā„ƒ)       -0.019672
StdGateHydraulicDiameter   -0.078172
NumberOfGates              -0.094894
CavitySurfaceToVolume      -0.435327
OverallSurfaceToVolume     -0.443351
NumberOfCavities           -0.457310
Name: Weight (g), dtype: float64
InĀ [13]:
del_features = weight_corr[weight_corr.abs() < 0.8]
del_features.keys()
Out[13]:
Index(['Fill time (sec)', 'Melt temperature (ā„ƒ)', 'Mold temperature (ā„ƒ)',
       'Packing pressure (MPa)', 'Packing time (sec)',
       'OverallProjectionAreaZX', 'MinGateHydraulicDiameter', 'StdFlowLength',
       'RunnerSurfaceArea', 'StdGateHydraulicDiameter', 'MinFlowLength',
       'MaxFlowLengthToThickness', 'MaxPartThickness', 'AvgFlowLength',
       'AvgFlowLengthToThickness', 'NumberOfGates', 'OverallProjectionAreaXY',
       'MaxFlowLength', 'RunnerVolume', 'StdFlowLengthToThickness',
       'MaxGateHydraulicDiameter', 'MinFlowLengthToThickness',
       'OverallProjectionAreaYZ', 'StdPartThickness', 'NumberOfCavities',
       'AvgPartThickness', 'AvgGateHydraulicDiameter', 'Overall_Compactness',
       'Cavity_Compactness', 'Runner_Compactness', 'CavitySurfaceToVolume',
       'RunnerSurfaceToVolume', 'OverallSurfaceToVolume'],
      dtype='object')
InĀ [14]:
train_x_fs = train_x.drop(list(del_features.keys()[5:]), axis = 1)
test_x_fs = test_x.drop(list(del_features.keys()[5:]), axis = 1)

print('train_x_fs: {}, train_y: {}'.format(train_x_fs.shape, train_y.shape))
print('test_x_fs: {}, test_y: {}'.format(test_x_fs.shape, test_y.shape))
train_x_fs: (3500, 9), train_y: (3500,)
test_x_fs: (100, 9), test_y: (100,)

2.3. Linear Regression Modeling¶

InĀ [15]:
from sklearn.linear_model import LinearRegression

reg = LinearRegression().fit(train_x_fs, train_y)

print('Regression Coefficient: \n{}\n'.format(reg.coef_))
print('Regression Bias: \n{}'.format(reg.intercept_))

pred_s_lr = reg.predict(test_x_fs)
plot_(pred_s_lr, test_y, 'Linear Regression with FS')
Regression Coefficient: 
[ 0.76727    -0.03221696 -0.0250693   0.00341566  0.72655702 -0.04489469
  0.00220443  0.00469149  0.77575597]

Regression Bias: 
5.3092896633051865

2.4. Deep Neural Networks (Multi-Layer Perceptron)¶

  • Raw dataset
  • Feature selected dataset



InĀ [16]:
tf.random.set_seed(42)

model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape = (train_x.shape[1],)),
    tf.keras.layers.Dense(units = 10, activation = 'relu'),
    tf.keras.layers.Dense(units = 10, activation = 'relu'),
    tf.keras.layers.Dense(units = 10, activation = 'relu'),
    tf.keras.layers.Dense(units = 10, activation = 'relu'),
    tf.keras.layers.Dense(units = 10, activation = 'relu'),
    tf.keras.layers.Dense(units = 1, activation = None)
])
InĀ [17]:
model.summary()
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 10)                380       
                                                                 
 dense_1 (Dense)             (None, 10)                110       
                                                                 
 dense_2 (Dense)             (None, 10)                110       
                                                                 
 dense_3 (Dense)             (None, 10)                110       
                                                                 
 dense_4 (Dense)             (None, 10)                110       
                                                                 
 dense_5 (Dense)             (None, 1)                 11        
                                                                 
=================================================================
Total params: 831
Trainable params: 831
Non-trainable params: 0
_________________________________________________________________
InĀ [18]:
model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate = 0.001),
              loss = 'mse')
InĀ [19]:
loss = model.fit(train_x, train_y, epochs = 50)
Epoch 1/50
110/110 [==============================] - 1s 3ms/step - loss: 1648.0643
Epoch 2/50
110/110 [==============================] - 0s 3ms/step - loss: 98.9756
Epoch 3/50
110/110 [==============================] - 0s 3ms/step - loss: 14.5452
Epoch 4/50
110/110 [==============================] - 0s 3ms/step - loss: 11.4503
Epoch 5/50
110/110 [==============================] - 0s 3ms/step - loss: 8.5021
Epoch 6/50
110/110 [==============================] - 0s 3ms/step - loss: 7.1835
Epoch 7/50
110/110 [==============================] - 0s 3ms/step - loss: 6.3429
Epoch 8/50
110/110 [==============================] - 0s 3ms/step - loss: 5.7080
Epoch 9/50
110/110 [==============================] - 0s 3ms/step - loss: 5.0012
Epoch 10/50
110/110 [==============================] - 0s 3ms/step - loss: 4.5427
Epoch 11/50
110/110 [==============================] - 0s 3ms/step - loss: 4.2173
Epoch 12/50
110/110 [==============================] - 0s 3ms/step - loss: 3.4958
Epoch 13/50
110/110 [==============================] - 0s 3ms/step - loss: 3.4844
Epoch 14/50
110/110 [==============================] - 0s 3ms/step - loss: 3.0655
Epoch 15/50
110/110 [==============================] - 0s 3ms/step - loss: 3.1229
Epoch 16/50
110/110 [==============================] - 0s 3ms/step - loss: 3.1040
Epoch 17/50
110/110 [==============================] - 0s 3ms/step - loss: 3.0606
Epoch 18/50
110/110 [==============================] - 0s 3ms/step - loss: 2.2575
Epoch 19/50
110/110 [==============================] - 0s 3ms/step - loss: 2.2680
Epoch 20/50
110/110 [==============================] - 0s 3ms/step - loss: 2.0988
Epoch 21/50
110/110 [==============================] - 0s 4ms/step - loss: 2.4578
Epoch 22/50
110/110 [==============================] - 0s 4ms/step - loss: 2.3471
Epoch 23/50
110/110 [==============================] - 0s 4ms/step - loss: 1.8365
Epoch 24/50
110/110 [==============================] - 0s 3ms/step - loss: 2.3753
Epoch 25/50
110/110 [==============================] - 0s 3ms/step - loss: 1.9287
Epoch 26/50
110/110 [==============================] - 0s 3ms/step - loss: 2.0505
Epoch 27/50
110/110 [==============================] - 0s 3ms/step - loss: 1.6447
Epoch 28/50
110/110 [==============================] - 0s 3ms/step - loss: 1.5820
Epoch 29/50
110/110 [==============================] - 0s 3ms/step - loss: 2.2741
Epoch 30/50
110/110 [==============================] - 0s 4ms/step - loss: 2.2164
Epoch 31/50
110/110 [==============================] - 1s 5ms/step - loss: 1.5752
Epoch 32/50
110/110 [==============================] - 0s 4ms/step - loss: 1.4708
Epoch 33/50
110/110 [==============================] - 0s 4ms/step - loss: 1.8566
Epoch 34/50
110/110 [==============================] - 0s 4ms/step - loss: 1.9300
Epoch 35/50
110/110 [==============================] - 1s 6ms/step - loss: 1.3955
Epoch 36/50
110/110 [==============================] - 0s 4ms/step - loss: 1.6484
Epoch 37/50
110/110 [==============================] - 0s 4ms/step - loss: 1.6280
Epoch 38/50
110/110 [==============================] - 0s 4ms/step - loss: 1.2482
Epoch 39/50
110/110 [==============================] - 0s 4ms/step - loss: 2.2573
Epoch 40/50
110/110 [==============================] - 1s 5ms/step - loss: 1.5343
Epoch 41/50
110/110 [==============================] - 1s 5ms/step - loss: 1.5565
Epoch 42/50
110/110 [==============================] - 0s 4ms/step - loss: 1.6434
Epoch 43/50
110/110 [==============================] - 1s 5ms/step - loss: 1.4292
Epoch 44/50
110/110 [==============================] - 0s 4ms/step - loss: 1.5619
Epoch 45/50
110/110 [==============================] - 1s 6ms/step - loss: 1.6500
Epoch 46/50
110/110 [==============================] - 1s 5ms/step - loss: 1.4675
Epoch 47/50
110/110 [==============================] - 1s 6ms/step - loss: 1.1956
Epoch 48/50
110/110 [==============================] - 1s 6ms/step - loss: 1.6599
Epoch 49/50
110/110 [==============================] - 1s 5ms/step - loss: 1.0315
Epoch 50/50
110/110 [==============================] - 1s 5ms/step - loss: 1.3436
InĀ [20]:
pred_dnn = model.predict(test_x)
plot_(pred_dnn, test_y, 'Deep Neural Network')
4/4 [==============================] - 0s 3ms/step
InĀ [21]:
tf.random.set_seed(42)

model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape = (train_x_fs.shape[1],)),
    tf.keras.layers.Dense(units = 10, activation = 'relu'),
    tf.keras.layers.Dense(units = 10, activation = 'relu'),
    tf.keras.layers.Dense(units = 10, activation = 'relu'),
    tf.keras.layers.Dense(units = 10, activation = 'relu'),
    tf.keras.layers.Dense(units = 10, activation = 'relu'),
    tf.keras.layers.Dense(units = 1, activation = None)
])

model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate = 0.001),
              loss = 'mse')

loss = model.fit(train_x_fs, train_y, epochs = 50)
Epoch 1/50
110/110 [==============================] - 1s 3ms/step - loss: 2128.4236
Epoch 2/50
110/110 [==============================] - 0s 3ms/step - loss: 272.1700
Epoch 3/50
110/110 [==============================] - 0s 3ms/step - loss: 230.1623
Epoch 4/50
110/110 [==============================] - 0s 3ms/step - loss: 156.3237
Epoch 5/50
110/110 [==============================] - 0s 3ms/step - loss: 72.3822
Epoch 6/50
110/110 [==============================] - 0s 3ms/step - loss: 16.6844
Epoch 7/50
110/110 [==============================] - 0s 3ms/step - loss: 5.0298
Epoch 8/50
110/110 [==============================] - 0s 3ms/step - loss: 3.8953
Epoch 9/50
110/110 [==============================] - 0s 3ms/step - loss: 3.1754
Epoch 10/50
110/110 [==============================] - 0s 3ms/step - loss: 2.5970
Epoch 11/50
110/110 [==============================] - 0s 3ms/step - loss: 2.3744
Epoch 12/50
110/110 [==============================] - 0s 3ms/step - loss: 2.0501
Epoch 13/50
110/110 [==============================] - 0s 3ms/step - loss: 1.8823
Epoch 14/50
110/110 [==============================] - 0s 3ms/step - loss: 1.7180
Epoch 15/50
110/110 [==============================] - 0s 3ms/step - loss: 1.6869
Epoch 16/50
110/110 [==============================] - 0s 3ms/step - loss: 1.4859
Epoch 17/50
110/110 [==============================] - 0s 3ms/step - loss: 1.8135
Epoch 18/50
110/110 [==============================] - 0s 3ms/step - loss: 1.4075
Epoch 19/50
110/110 [==============================] - 0s 3ms/step - loss: 1.3985
Epoch 20/50
110/110 [==============================] - 0s 3ms/step - loss: 1.5828
Epoch 21/50
110/110 [==============================] - 0s 3ms/step - loss: 1.4436
Epoch 22/50
110/110 [==============================] - 0s 3ms/step - loss: 1.3951
Epoch 23/50
110/110 [==============================] - 0s 3ms/step - loss: 1.5282
Epoch 24/50
110/110 [==============================] - 0s 3ms/step - loss: 1.3439
Epoch 25/50
110/110 [==============================] - 0s 3ms/step - loss: 1.6252
Epoch 26/50
110/110 [==============================] - 0s 3ms/step - loss: 1.5180
Epoch 27/50
110/110 [==============================] - 0s 3ms/step - loss: 1.1968
Epoch 28/50
110/110 [==============================] - 0s 3ms/step - loss: 1.3851
Epoch 29/50
110/110 [==============================] - 0s 3ms/step - loss: 1.4960
Epoch 30/50
110/110 [==============================] - 0s 3ms/step - loss: 1.2361
Epoch 31/50
110/110 [==============================] - 0s 3ms/step - loss: 1.4313
Epoch 32/50
110/110 [==============================] - 0s 3ms/step - loss: 1.3130
Epoch 33/50
110/110 [==============================] - 0s 3ms/step - loss: 1.9432
Epoch 34/50
110/110 [==============================] - 0s 3ms/step - loss: 1.1492
Epoch 35/50
110/110 [==============================] - 0s 3ms/step - loss: 1.0483
Epoch 36/50
110/110 [==============================] - 0s 3ms/step - loss: 1.1009
Epoch 37/50
110/110 [==============================] - 0s 3ms/step - loss: 1.0038
Epoch 38/50
110/110 [==============================] - 0s 3ms/step - loss: 1.0490
Epoch 39/50
110/110 [==============================] - 0s 3ms/step - loss: 1.2210
Epoch 40/50
110/110 [==============================] - 0s 3ms/step - loss: 1.8674
Epoch 41/50
110/110 [==============================] - 0s 3ms/step - loss: 1.4005
Epoch 42/50
110/110 [==============================] - 0s 3ms/step - loss: 1.6804
Epoch 43/50
110/110 [==============================] - 0s 3ms/step - loss: 1.2652
Epoch 44/50
110/110 [==============================] - 0s 3ms/step - loss: 1.3613
Epoch 45/50
110/110 [==============================] - 0s 3ms/step - loss: 1.1826
Epoch 46/50
110/110 [==============================] - 0s 3ms/step - loss: 1.0808
Epoch 47/50
110/110 [==============================] - 0s 3ms/step - loss: 1.0203
Epoch 48/50
110/110 [==============================] - 0s 3ms/step - loss: 1.3558
Epoch 49/50
110/110 [==============================] - 0s 3ms/step - loss: 0.9993
Epoch 50/50
110/110 [==============================] - 0s 3ms/step - loss: 1.0943
InĀ [22]:
pred_s_dnn = model.predict(test_x_fs)
plot_(pred_s_dnn, test_y, 'Deep Neural Network with FS')
4/4 [==============================] - 0s 3ms/step
InĀ [23]:
plt.figure(figsize = (15, 20))
plt.subplot(321)
plot_(pred_dt, test_y, 'Decision Tree', False)
plt.subplot(322)
plot_(pred_rf, test_y, 'Random Forest', False)
plt.subplot(323)
plot_(pred_lr, test_y, 'Linear Regression', False)
plt.subplot(324)
plot_(pred_s_lr, test_y, 'Linear Regression with FS', False)
plt.subplot(325)
plot_(pred_dnn, test_y, 'Deep Neural Network', False)
plt.subplot(326)
plot_(pred_s_dnn, test_y, 'Deep Neural Network with FS', False)
InĀ [3]:
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')